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(54) Imaging system with alterable local convolution kemal 

(57) A method for innage processing combining a 
device-specific image processing kernel operation with 
a general image processing kernel operation. Device 
specific parameters are sent from a host computer 
(202) to an imaging device (200). The imaging device 
selects appropriate device-specific convolution coeffi- 
cients. The host computer selects additional kernel 
operations. In a first example embodiment, the device- 
specific coefficients are then uploaded (306) to the host 
conputer and the host computer convolves (310) the 
host-specified operations with the device-specific oper- 
ations. The combined kernel is then downloaded (312) 
to the imaging device. In an alternative embodiment, the 
host-specified operations are downloaded to the imag- 
ing device for combining. The combined kernels are 
then used by the imaging device for convolution opera- 
tions (31 4} on an image. As a result host computer soft- 
ware can modify a kernel operation within an imaging 
device with minimal knowledge of device-specific 
parameters such as native resolution. 
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Description 

FIELD OF INVEhmON 

5 This invention relates generally to computerized image processing and nrore specifically to an system in which an 
imaging device Includes programmable convolution hardware and in which convolution kernel values local to the Imag- 
ing device are modified in response to options selected by a host computer operator. 

BACKGROUND OF THE INVENTION 

10 

Digital cameras, image scanners, copiers, facsimile machines and other electronic imaging devices provide images 
in an electronic form suitable for transmission, printing or storage. Picture elements (pixels) are represented numerically 
as intensity values or coordinates in a color space. Once images are acquired in computer readable form, they may be 
"processed" to remove distortions, to reduce noise, to modify color, to accentuate outlines, and so forth. This patent 
15 document is concerned with image processing, and in particular, spatial fitters, also called kernel operations or convo- 
lution kernels. 

Photosensor an'aysused in imaging devices typically have thousands of individual photosensitive elements. Each 
photosensitive element, in conjunction with optics assemblies, measures light intensity from an effective area drfining 
a pixel on the image being scanned. The resolution defined by the optics and the individual photosensitive elements is 
20 called the "native" resolution. Additional pixels may be inserted by interpolation or padding, creating a higher "effective" 
resolution. Lower resolutions than the effective resolution may then be obtained by dropping selected pixels, called dec- 
imation. 

Images have spatial frequency information. For example, patterns of dots in half-tone printed images have a spatial 
frequency. Periodic sampling of spatial patterns may produce aliasing if the spatial sampling frequency is less than 

25 twice the highest spatial frequency in the image being sampled. In particular, decimation to provide a tower resolution 
may introducing aliasing artifacts if the digitized image is not appropriately filtered. Therefore, images nnay need to be 
spatially low-pass filtered before decinration is performed. 

Another common image processing filter of interest is spatial high-pass filtering, and in particular Laplacian opera- 
tors. Visually. Laplacian operators improve image contrast at edges, making edges easier for the viewer to see. making 

30 an image appear sharper. 

Both spatial low pass filtering and spatial high pass filtering may be computed by spatial convolution, in which a 
pixel value and its surrounding neighboring pixel values are multiplied by coefficients, the results summed, and the 
resulting value replacing the original pixel value. Convolution in the spatial domain is mathematically equivalent to mul- 
tiplication in the frequency domain and linear filtering can be implemented in either domain. 

35 For a general tutorial overview of image processing, see, for example. Ross, John C. The Image Processing Hand- 
book. 2nd Edition (1995), ORG Press, Inc. Low pass spatial filtering is discussed In Ross at pages 155-164. Laplacian 
operators are discussed in Ross at pages 225-232. 

Electronic imaging devices are often connected to a host computer. The host computer may include control soft- 
ware for the imaging device, enabling a computer operator to change parameters within the imaging device. In general, 

40 for computer control of peripheral devices, it is often preferable for host software to be device-independent. That is, host 
software should not have to be concemed with specific hardware specifications in a peripheral device. There is a need 
for an imaging system in which device-specific operations can be perfonmed within an imaging device, transparent to 
the host computer, and in which a host computer operator can specify some control in a device-independent manner. 

45 SUMMARY OF THE INVENTION 

An electronic imaging device includes a programmable convolution kerne! with hardware computation within the 
imaging device. Given a resolution specified by a host computer, the imaging device interpolates or pads between 
native pixels to provide pixels at a higher effective resolution, and then low-pass filters the higher resolution image and 

so decimates to provide the specified resolution. The imaging device selects filter coefficients appropriate for the device- 
specific native resolution and the effective resolution. In addition, an operator can specify image sharpening or other 
spatial filtering. The host-specified Laplacian kernel (or other spatial filter) is then convolved with the device-specific 
convolution kernel, with appropriate modification for decimation. In one example embodiment, the host computer 
uploads ttie device-dependent convolution kernel from the imaging device, convolves a host-specified Laplacian kennel 

55 with the uploaded convolution kernel, arxi downloads the resulting modified convolution kernel to ttie imaging device. 
In an alternative example emtxxliment the host-specified Laplacian kernel is downloaded to the peripheral imaging 
device and tiie imaging device convolves the two kernels. In either embodiment, after combining kernels, the inriaging 
device tiien computes combined device-dependent and host-specified spatial filtering, with ttie host required to have 
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only minimal knowledge of native resolution or other device specific parameters. 



BRIEF DESCRIPTION OF THE DRAWINGS 



5 Rgure 1 A is a pbt of pixel (intensity) values as a function of column number for one row of an example image. 
Rgure 1 B is a pbt of the pixel values of figure 1 A after being modified by a spatial low-pass filter. 
Rgure 1C is a plot of the pixel values of figure 1 A after being modified by a spatial high-pass filter. 
Rgure 2 is a block diagram of an imaging system. 
Rgure 3 is a flow chart of a method in accordance with the invention. 

10 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT OF THE INVENTION 



The following array of numbers represents pixel values P(i,j) in a two dimensional image. Assume for illustration 
that pixels having low numerical values are dark and pixels having high numerical values are light. Figure 1 A is a graph 
IS of row number 2 (r=2) as a function of column number Q). 
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The present patent document is concerned with spatial filtering using convolution. The following example illustrates 
a convolution kernel. 

30 

W(i-1 ,j) 
W(i,j-1) W(i,j) W(i,j + 1) 
W(i+ 1 ,j) 

35 



For each pixel POJ), the convolution computation is as follows: 

P(i, j) = (W(i, j)*P(i, j)+W{i+i, j)*P(i+l, j)+W(i, j-l)*P(i, j-1) 
+W(i,j + l)*P(i,j + 1) +W(i-l . ,j)*P(i-l / 
[W(i, j)+W(i+l, j)+W(i, j-l)+W(i, j+l)+W(i-l, j) ] 



Convolution operations are destructive, so a copy of each original numerical pixel value must be stored long 
enough to ensure that the computation is performed on original pixel values, not modified pixel values. In addition, spe- 
50 cial rules are required for the outer edges of the image, by creating artificial boundaries or by choosing to not perform 
the filtering operation for pixels adjacent to a boundary. In general, convolution kernels are a matrix of coefficients. In 
the above example, the matrix is 3x3 with the corner coefficients having a value of zero. Much larger matrices may be 
used. 

Consider the folbwtng convolution kernel which is an example of a spatial low-pass fitter: 



55 
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1 

12 1 

5 1 

For each pixel P(i.j). the computation is as follows: 

w 

P(i.j)=[2*P(IJ)+P(i+ljKP(i.MKP(i.j+l)+P(i-l.D]/6 

For example, consider P(2,5) highlighted above. 

15 P(2.5) = [2*P(2.5)+P(1.5)+P(2,4)+P(2.6)+P(3.5)]/6 = [40+20+1 a+20+20]/6= 18.3 

As a result of applying the convolution operation, the value of P(2.5) Is changed from 20 to 18.3. The following num- 
bers (rounded to integers) represent row 2 above after application of the above spatial low-pass filter (edge pixels are 
not filtered). 

20 

i=2 10 10 10 12 18 20 20 22 28 30 30 32 38 40 40 
j= 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 

25 

Rgure 1B illustrates the above tittered row. Reference number 100 in figure IB designates P(2,5) used for the 
above numerical example. Note that the abrupt steps in intensity now have a slight spatial transition. 
30 Now consider the following convolution kernel, which is an example of a Laplacian operator, a spatial high-pass fil- 
ter: 



-1 



35 




40 For each pixel P(i.j), the computation is as follows: 

PCi.j) = [(6)*P(i.j)-P(i + 1.i)-P(i.i-1)-PCr.i + 1)-P(i-1.j)l/2 

The following numbers (rounded to integers) represent original row 2 after application of the above Laplacian oper- 
45 ator (the edge pixels are not filtered). 

i=2 10 10 10 5 25 20 20 15 35 30 30 25 45 40 40 
j= 1 2 3 4 5 6 . 7 8 9 10 11 12 13 14 15 



Rgure 1C is a graph of original row number 2 above after application of the above Laplacian operator. Note that 
55 steps in intensity in the originaJ image (figure 1 A) result in an undershoot followed by an ove^hoot In the filtered image 
(figure 1C). As a result steps in intensity having an increment of 10 in figure 1 A have an inaement of 20 in figure 1C. 
That is, intensity steps or edges or boundaries are exaggerated. The above filters are used for illustration only. In an 
actual implementation, tfie cut-off spatial frequencies of the low-pass filter and the sharpening filter must be consistent. 
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"mat is. the sharpening fitter should exaggerate spatial frequencies that are just below the cut-off frequency for the low- 
pass filter. 

Rgure 2 illustrates an imaging system. An imaging device 200 is connected to a host computer 202. The imaging 
device 200 includes a controller 204 with special hardware for rapidly computing convolution operations as illustrated 

5 abova If the imaging device 200 has a two-dimensional photosensor an^y, for example a digital camera, the effective 
resolution in both dimensions may be implemented by interpolation or padding followed by decimation. If the imaging 
device 200 has a one-dimensional photosensor array, the effective resolution in one dimension may be varied by vary- 
ing the scanning speed. In a specific example embodiment, device 200 is a scanner and the convolution hardware 
implements a general purpose 9x9 convolution kernel (or smaller) which is expanded by padding to 72x9 as discussed 

10 below. 

In general, a spatial low pass fSter may be optimized for a specific cut-off frequency, expressed as pixels per inch 
(or mm) or lines per inch (or mm). Therefore, an optinial spatial low pass fitter varies with the native resolution and the 
effective resolution of the imaging device. In addition to native resolution and effective resolution, there are device-spe- 
cific specifications such as the modulation transfer function that may affect a choice for low-pass fitter parameters. In 

15 the specific example embodiment, scanners have a native resolution of 300 pixels per inch (12 pixeb per mm) as meas- 
ured on the image being scanned. For a requested resolution in the X dimension that is other than the native resolution, 
the image is padded with zeros to generate an 8X effective resolution of 2.400 pixels per inch (95 pixels per mm). Then, 
for a requested X-dimension resolution less than the effective resolution, device 200 selects a low pass filter that will 
attenuate at least any spatial frequency greater than twice the requested resolution. The appropriate coefficients are 

20 placed in the local 9x9 convolution matrix (padded with zeros to form a 72x9 matrix) . The fitter output is at the requested 
resolution. That is. decimation is an integral part of the fitter computation. Note again that the low-pass filter coefficients 
are specific to the imaging device 200. These filter coefficients could be selected by software in the host computer 202, 
but this would require host 202 to know specific hardware specifications for imaging device 200. In order to keep the 
software in the host computer 202 as general as possible, it is preferable to make the host computer software independ- 

25 ent of imaging device specifications. Then, various imaging devices may be used with different internal specifications 
witiiout requiring changes in the host computer software. 

The host computer 202 may also specify certain convolution operations. For example, for binary images such as 
line drawings, image "sharpening" or high-pass filtering is desirable to provide sharp edges. In general, convolution 
operations are linear and can be combined. For example, a low-pass filter for decimation may be combined with a high- 

30 pass fitter for edge sharpening. With specific hardware 204 available in imaging device 200 for convolution computation, 
it is desirable to use that hardware for a Laplacian operation or other operator specified imaging processing operations. 
The problem then is to conrtbine a host-selected Laplacian operation witii a imaging-device-selected low pass filter. In 
a first example emt)odiment, the host 202 uploads the padded convolution mati'ix from the imaging device 200, applies 
a host-selected filter to ttie convolution matrix, and then downloads the resulting matrix back to the imaging device for 

35 use on a scanned image. As a specific example, the host 202 uploads a 9x9 matrix of coefficients (padded to 72x9) 
from imaging device 200, applies a padded Laplacian operator to the 72x9 matrix of coefficients just as if the coeffi- 
cients were pixel values in the earlier examples above, and then downloads the resulting 72x9 matrix back to the imag- 
ing device 200. 

When convolving the two kernels, the host-specified kernel coefficients and the device-specified kernel coefficients 
40 must be based on the same resolution. Recall that the filter coefficients in the imaging device 200 are based on ded- 
malion of an effective resolution (for one dimension only in tiie case of a scanner). In the specific example implemen- 
tation discussed above, the effective resolution is eight times higher than the native resolution. This may also be device- 
spedf ic. Therefore, in addition to the device-specific fitter coefficients, tiie imaging device must also communicate infor- 
mation regarding interpolation (or padding) or decimation. In a specific embodiment in which the kernels are convolved 
45 by the host computer, the host also uploads a decimation factor (an integer) that essentially specifies that tiie host 
Laplacian kernel co^icients must be "padded" by additional coefficients having a value of zero. For example, assuming 
tiiat the peripheral imaging device pads by 8X, each pair of host Laplacian kernel coefficients must be separated by 7 
coefficients having a value of zero so that tfie host kernel resolution is the same as the imaging device kernel resolution. 
For an imaging device having a two^imensional imaging array, the Laplacian kernel coefficients must be padded in 
50 both the X and Y dimensions. For an imaging device having a one-dimensional photosensor array the Laplacian kemel 
only needs to be padded in a single dimension. Rnally, note that in general, X and Y resolutions may be different. There- 
fore, in general, the imaging device may have to send two decimation factors, one for X resolution and one for Y reso- 
lution. 

As an alternative embodiment, tiie host may download the host-specified Laplacian kemel to the peripheral imag- 
55 ing device and the imaging device may then convolve tiie two kernels. Recall that the host also specifies resolution, so 
no separate informaton is required by the imaging device to determine how to pad the laplacian kernal witii additional 
coefficients having a value of zero. Therefore, in the altemative embodiment, the host software can be conrtpletely 
device-independent, whereas in the first embodiment, the imaging device must tell the host some information about 
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effective resolution. 

Figure 3 is a flow chart of a method in accordance with the first example embodiment. At step 300. an operator 
selects various parameters such as resolution and sharpening. Note that resolution and sharpening are just examples, 
and that there are other convolution operations that may be appropriate for host-selection of coefficients or imaging 

5 device selection of coefficients. At step 302, resolution (or other parameters appropriate for image de/ice selection of 
coefficients) ts sent to the imaging device. At step 304, software within the imaging device selects appropriate convolu- 
tion coefficients based on native and effective resolution of the imaging device. At step 306, the imaging-device- 
selected coefficients are uploaded to the host computer. At step 308, the host computer selects appropriate convolution 
coeffients for sharpening or other device independent parameters. At step 310. software in the host computer convolves 

10 the host-selected coefficients with the coefficients uploaded from the imaging device. At step 312. the combined coef- 
ficients are downloaded to the imaging device. At step 314, software in the imaging device uses special hardware in the 
imaging device to perform the convolution operation on an image. Again, in an alternative embodiment, at steps 306- 
310, the host computer may download host-selected coefficients to the imaging device and the imaging device then 
convolves the two kemals. 

IS Note that the sample filters above are symmetrical about the center row and column. In the specific example 
embodiment, an odd number of rows and columns is assumed and symmetry atxjut the center row and column is 
assumed so that only one corner of a convolution matrix is actually stored. That is. for a 3x3 convolution matrix, only 
2x2 matrix needs to be stored, for a 5x5 convolution matrix only a 3x3 matrix needs to be stored and so forth. In the 
specific first example embodiment, for a 3x3 Laplacian, a boundary of two elements is added to each edge of the 

20 uploaded convolution matrix with all coefficients in the boundary having a value of zero. If the Laplacian kernal is greater 
than 3x3. a larger boundary must be added. 

The foregoing description of the present invention has been presented for purposes of illustration and description. 
It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and other modifications and 
variations may be possible in light of the above teachings. The embodiment was chosen and described in order to best 

25 explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize 
the invention in various embodiments and various modifications as are suited to the particular use contemplated. It is 
intended that the appended claims be construed to include other alternative embodiments of the invention except inso- 
far as limited by the prior art. 

30 Claims 

1 . A method of forming a convolution kernal in an imaging system, the image system comprising a host computer 
(202) and an imaging device (200), the method comprising the following steps: 

35 (a) sending (302) an imaging parameter from the host computer to the imaging device; 

(b) selecting (304), by the imaging device, first convolution coefficients based on the imaging parameter from 
the host computer; 

(c) selecting (308), by the host computer, second convolution coefficients; 

(d) convolving (310) the second convolution coefficients with the first convolution coefficients to form third con- 
40 volution coefficients; and 

(f) convolving (314), by the imaging device, the third convolution coefficients with an image. 

2. The method of claim 1 where the imaging parameter is resolution. 

45 3. The method of claims 1 or 2 where the first convolution coefficients form a kernal for spatial low-pass filtering. 

4. The method of any one of the preceding claims where the second convolution coefficients form a Laplacian oper- 
ator. 

50 5. The method of any one of the preceding claims further comprising the following step before step (d): 

(d1) uploading (306), by the host computer, from the imaging device, the first convolution coefficients; 

and further comprising the following step after step (d): 
(d2) downloading (312), by the host computer, to the imaging device, the third convolution coefficients. 

55 

6. The method of any one of the preceding claims further comprising the following step before step (d): 

(d1) downloading, by the host computer, to the imaging device, the second convolution coefficients. 
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7. An imaging system, the image system comprising a host computer (202) and an imaging device (200), and indud- 



(a) means for sending (302) an imaging parameter from the host computer to the imaging device; 

(b) means for selecting (304). by the imaging device, first convolution coefficients based on the imaging param- 
eter from the host computer; 

(c) means for selecting (308). by the host computer, second convolution coefficients; 

(d) means for convolving (310) the second convolution coefficients with the first convolution coefficients to form 
third convolution coefficients: and 

(f) means fa convolving (314). by the imaging device, the third convolution coefficients with an image. 

8. The system of claim 7 where the first convolution coefficients form a kemal for spatial tow-pass filtering. 

9. The system of claims 7 or 8 where the second convolution coefficients form a Lapladan operator. 

10. The system of any one of claims 7-9 further including: 

means for iploading (306), by the host computer, from the imaging device, the first convolution coefficients, 
before they are convolved to form the third convolution coefficients; and 

means for downloading (312), by the host computer, to the imaging device, the third convolution coefficients 
after they are convolved by the convolving means. 
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